Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformer

Movement synchrony reflects the coordination of body movements between interacting dyads. The estimation of movement synchrony has been automated by powerful deep learning models such as transformer networks. However, instead of designing a specialized network for movement synchrony estimation, previous transformer-based works broadly adopted architectures from other tasks such as human activity recognition. Therefore, this paper proposed a skeleton-based graph transformer for movement synchrony estimation. The proposed model applied ST-GCN, a spatial-temporal graph convolutional neural network for skeleton feature extraction, followed by a spatial transformer for spatial feature generation. The spatial transformer is guided by a uniquely designed joint position embedding shared between the same joints of interacting individuals. Besides, we incorporated a temporal similarity matrix in temporal attention computation considering the periodic intrinsic of body movements. In addition, the confidence score associated with each joint reflects the uncertainty of a pose, while previous works on movement synchrony estimation have not sufficiently emphasized this point. Since transformer networks demand a significant amount of data to train, we constructed a dataset for movement synchrony estimation using Human3.6M, a benchmark dataset for human activity recognition, and pretrained our model on it using contrastive learning. We further applied knowledge distillation to alleviate information loss introduced by pose detector failure in a privacy-preserving way. We compared our method with representative approaches on PT13, a dataset collected from autism therapy interventions. Our method achieved an overall accuracy of 88.98% and surpassed its counterparts by a wide margin while maintaining data privacy.Comment: Accepted by 24th ACM International Conference on Multimodal Interaction (ICMI'22). 17 pages, 2 figure

Dyadic Movement Synchrony Estimation Under Privacy-preserving Conditions

Movement synchrony refers to the dynamic temporal connection between the motions of interacting people. The applications of movement synchrony are wide and broad. For example, as a measure of coordination between teammates, synchrony scores are often reported in sports. The autism community also identifies movement synchrony as a key indicator of children's social and developmental achievements. In general, raw video recordings are often used for movement synchrony estimation, with the drawback that they may reveal people's identities. Furthermore, such privacy concern also hinders data sharing, one major roadblock to a fair comparison between different approaches in autism research. To address the issue, this paper proposes an ensemble method for movement synchrony estimation, one of the first deep-learning-based methods for automatic movement synchrony assessment under privacy-preserving conditions. Our method relies entirely on publicly shareable, identity-agnostic secondary data, such as skeleton data and optical flow. We validate our method on two datasets: (1) PT13 dataset collected from autism therapy interventions and (2) TASD-2 dataset collected from synchronized diving competitions. In this context, our method outperforms its counterpart approaches, both deep neural networks and alternatives.Comment: IEEE ICPR 2022. 8 pages, 3 figure

Floquet Chern Insulators of Light

Achieving topologically-protected robust transport in optical systems has recently been of great interest. Most topological photonic structures can be understood by solving the eigenvalue problem of Maxwell's equations for a static linear system. Here, we extend topological phases into dynamically driven nonlinear systems and achieve a Floquet Chern insulator of light in nonlinear photonic crystals (PhCs). Specifically, we start by presenting the Floquet eigenvalue problem in driven two-dimensional PhCs and show it is necessarily non-Hermitian. We then define topological invariants associated with Floquet bands using non-Hermitian topological band theory, and show that topological band gaps with non-zero Chern number can be opened by breaking time-reversal symmetry through the driving field. Furthermore, we show that topological phase transitions between Floquet Chern insulators and normal insulators occur at synthetic Weyl points in a three-dimensional parameter space consisting of two momenta and the driving frequency. Finally, we numerically demonstrate the existence of chiral edge states at the interfaces between a Floquet Chern insulator and normal insulators, where the transport is non-reciprocal and uni-directional. Our work paves the way to further exploring topological phases in driven nonlinear optical systems and their optoelectronic applications, and our method of inducing Floquet topological phases is also applicable to other wave systems, such as phonons, excitons, and polaritons

Birth of strange nonchaotic attractors in a piecewise linear oscillator

ACKNOWLEDGMENTS We sincerely thank the people who gave valuable comments. This paper was supported by the National Natural Science Foundation of China (NNSFC) (No. 12172306) and the Innovation Fund Project of Colleges and Universities in Gansu Province (No. 2021A-040).Peer reviewedPostprin